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ABSTRACT 

A literature review examined how literacy programs 
can assess their ef f ectiveness by looking at outcomes and using this 
information. Very few empirical studies showed the impacts of 
literacy programs or identified predictors of program success. One 
reason was the lack of common criteria for evaluating performance. 

One evaluation issue was whether the same measures should be used to 
assess both individual progress and program ef f ectiveness . Results of 
the U.S. National Evaluation of Adult Education Programs (NEAEP) 
showed that using the same types of measures across programs allowed 
discovery of what program components most affected outcomes. The 
NAEAP study was also used to investigate research on variables 
assumed to be linked to program ef f ectiveness . According to the 
study, the strongest predictors of retention were support services, 
instruction during the day, and type of learning environment. Age was 
not a barrier to acquiring literacy and increased hours of 
instruction were not sufficient to ensure increased skill 
development. Literacy skills were not easily transferred between 
workplace and general literacy programs; family literacy programs had 
potential for both adult and child literacy. Questions for further 
outcomes“bas ed research were suggested, including the cost benefit of 
putting limited resources into instruction rather than outreach, 
maintenance of learner motivation, and instructor training. (Contains 
31 references.) (YLB) 
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Executive Summary 



Introduction 

This report looks at issues and research associated with demonstrating the 
effectiveness of adult literacy programs. The report is divided into three 
sections. Section One discusses the need to sissess program outcomes. 

Section Two reviews research on variables affecting the outcomes of literacy 
programs. Section Three provides the report’s concliisions and 
recommendations. 

Section One 

There are very few empirical studies that show the impacts of literacy 
programs or identify predictors of program success. One reason, among 
many, for the lack of hard data on outcomes, is the lack of common criteria 
for evaluating performance. Literacy educators are likely to emphasize 
process rather than results; consequently, knowledge and ideas about 
literacy practice stem primarily from shared values and anecdotal evidence. 
This report makes the case that literacy practitioners, students and policy 
makers all would benefit from reliable data on outcomes. 

Section One looks at approaches to the evaluation of programs and student 
learning, and recent work that has been done in the area of outcome-based 
education. Results of the U.S. National Evaluation of Adult Education 
Programs (NEAEP) study, a recent large scale study of programs, are 
reported. 

Section Two 

This section looks at research on variables which have been assumed to be 
linked to program effectiveness, including; student retention and attendance, 
the amount of time it takes adults to acquire or improve their literacy and 
the impacts of program structure and design on different types of learners. 
The NEAEP study is a major source of information in this section. 

According to the NEAEP study, the strongest predictors of retention are: 

• the presence of support services (such as counseling); 

• instruction during the day when more time is available to learn; and 

• the type of learning environment (such as participation in a learning lab or independent 
■ :Study in addition to classroom learning). 
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As for learning time, the study found that: 

• age is not a barrier to acquiring literacy. Adult learners are as adept at acquiring 
literacy as children but it takes longer. They may, however, acquire fluency more 
slowly than children. 

• increasing hours of instruction is not sufficient to assure increased skill development. 
This is further complicated by the need for practice after skills are acquired. 

Regarding program structure and design, the NEAEP smdy found: 

• program goals are easier to define for more targeted, “contextualized” programs, such 
as family literacy or workplace programs. The practicality of such programs is also 
appealing to funders (i.e. employers, school districts). 

• literacy skills are not easily transferred between workplace literacy programs, for 
example, and general literacy programs. This fact has led an increasing number of 
funding agencies and providers to develop targeted programs. 

• targeted programs are motivating because of a direct relevance to the learner’s 
situation (i.e. the promise of a job, a promotion or some other recognition). 

• the amount of time needed for skill improvement may be decreased by half when 
targeted programs use materials learners use in everyday work. 

• family literacy programs, using children’s books, etc., appear to have great potential 
for both adult and child literacy. There is high motivation and natural opportunities to 
practice skills acquired. The need for more evaluation research on family literacy is 
noted. 

Finally the report addresses other factors that can affect learners. These include: 
instructors, support services, program structure, program costs and learner characteristics 
themselves. Only one variable has been found to be “consistently and substantially” 
related to literacy gains for all types of learners: the learner’s own prior achievement level. 

Section Three 

This section poses a variety of questions which need to be answered based on further 
outcomes-based research, including: 

• Would there be more of an impact by putting limited resources into instruction instead 
of putting more effort and money into reaching out to the majority of those not getting 
literacy training? 

• How can literacy programs maintain learner motivation? 

• Should more targeted programs be offered instead of general literacy programs? 

• What kind of training should instructors have? 

• Under what circumstances does class/group size matter? 
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Preface 



The Ontario Literacy Coalition (OLC) is committed to improving levels of 
literacy among adults in Ontario. OLC believes that the strategy most likely 
to resxilt in increased adult literacy levels is to make adult literacy programs 
available, accessible, and effective. This report, ex aminin g issues and 
research related to assessing the effectiveness of adult literacy programs, was 
written to support the advocacy and field development work of the OLC as 
well as the work being done by other groups who share our commitment to 
adult literacy. 

Service providers face unprecedented challenges associated with sustaining 
services that rely on public funding. Deficit reduction is a top priority of both 
federal and provincial governments. The provincial government of Ontario is 
committed to stimulating private sector activity and reducing the size of the 
public sector. In this context, all programs in Ontario that rely on public 
funds are vulnerable to the possibility of drastic funding cuts or even 
elimination. Which programs will survive? How will adult literacy programs 
fare? What can literacy practitioners and the Ontario Literacy Coalition do 
to increase the survival odds of adult literacy programs in Ontario? 

Survival in the 1990’s 

To date, there has been no clear statement from government explaining the 
criteria being used to decide which programs to cut. When speculating about 
the future of literacy programs, the best we can do is: consider what we have 
learned so far from recent meetings with J/IPPs and senior bureaucrats and 
try to put ourselves in the shoes of the decision makers and think the way 
they do. When we do this, the survival of any government-funded program 
seems likely to be tied to three considerations: 

• the extent to which a program or service is viewed as befitting the role of 
government, 

• the extent to which a program or service targets a need which is 
recognized by government and is seen as being tightly linked to the 
government’s agenda', and 

• the extent to which the government believes that the program or service 
being delivered actualb. addresses and ultimately reduces the need, as this 
ffovemment understands that need. 

o 
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Measured against the first two of these criteria, the survival odds for adult 
literacy programs look pretty good - at least in the short term. The provincial 
government’s responsibility for basic education is clearly established in law, 
though its responsibility for adiilt students is not. In addition, many signs 
point to a devolution of responsibility for adiilt training from the federal 
government to the provinces. 

With respect to Ontario’s adult literacy issues, we have the impression that 
man y members of this government currently; 

• believe that there is a serious adiilt literacy problem in Ontario; and 

• believe that this problem is critically linked to the government’s 
commitments to reduce the deficit, stimulate the economy and get people 
back to work. 

One concern is that this government may subscribe to too narrow a view of 
the connections between literacy and the government’s priorities - focusing 
nearly exclusively on the links between literacy and labour force 
development issues. In all our contacts with decision makers, OLC 
highlights the equally vital connections between literacy and other issues on 
the government’s agenda. To help us do this, we have developed a series of 
fact sheets to document these links. For example, a fact sheet on literacy and 
health highlights research that shows increased health problems (and thus 
higher health care costs) for adults with low literacy skills. Another fact 
sheet on the intergenerational cycle of literacy highlights research on the 
impact of parents’ literacy levels on the academic performance of their 
children. Similar fact sheets have been developed showing the relationships 
between literacy and: crime and incarceration; safety and accidents; and the 
ability of seniors to live independently. Copies of these fact sheets may he 
obtained from OLC or from the nearest regional literacy network. 

A second concern regarding long-term survival for literacy programs is 
related to challenges associated with demonstrating the effectiveness of adult 
literacy programs. More than ever before, politicians and members of the 
public who are not directly involved in literacy work are challenging us to 
provide evidence that Ontario’s literacy programs increase the literacy skills 
of adults. 



Quality Standards Are a Start 



The Quality Standards Framework for Literacy Programs, developed in 
consultation with literacy practitioners across the province and adopted by 
the Board of Directors of the Ontario Training and Adjustment Board, 
provides cui important starting point for aissessing programs against a 
common yardstick. Two out of 18 quality standards call specifically for 
evaluation of student learning and/or program effectiveness (i.e. standard 8 
calls for on-going evaluation of learner’s progress; standard 18 states that 
the program will evaluate its effectiveness annually). Neither of these, 
however, explicitly calls for systematic measurement of increases in the 
literacy cind/or numeracy skills of students. Other standards reflect the 
conventional wisdom held in Ontario regarding features of effective literacy 
programs. 

This report suggests that reliable research on the links between these 
program features and actual student learning outcomes is needed before we 
can know whether the application of these standards actually correlates with 
the effectiveness of Ontario’s literacy programs. In the current climate of 
diminished resources for public programs, there is reason to believe that the 
pressure to demonstrate the impacts of our programs will increase rather 
than abate. The literacy field’s ability to offer such proof may have a major 
bearing on the long term survival prospects of adult literacy programs. 

This report has been written to stimulate thought and discussion about how 
literacy programs can assess their effectiveness by looking at outcomes, how 
they can use this information to improve the effectiveness of their programs, 
and how this information can be used to inform decision makers about the 
value of adult literacy programs. Recommendations in this report represent 
the author’s views and are not necessarily the position of the Ontario 
Literacy Coalition. 




Section One 



The Case for Outcome-Based Evaluation 
of Adult Literacy Programs 



1.1 The Need to Know Outcomes 



Does adult literacy education have a positive impact on 
both learners and society? While the absolute validity of 
even the best research on the question is limited, together 
the various impact studies present a preponderance of 
evidence to the affirmative. Yet there are still man y 
imanswered questions, and first and foremost among 
them may be the issue of impact itself. (Beder, p. 118). 



This report looks at what and how literacy programs know 
about their impacts or outcomes, and how this information can 
be used to support and improve services. The report is divided 
into three sections. Each section is further subdivided under 
topic headings. The first section includes this introduction and 
looks at issues surrounding the evaluation of literacy program 
outcomes. The second section reviews research correlating 
program outcomes with specific client or program variables. 
The third and final section provides conclusions, recom- 
mendations for further work in the area of literacy/ program 
evaluation, and lists references cited in the body of the report. 



What do we know about outcomes? 



Few empirical 
studies examine 
the impacts of 
adult literacy 
programs. 



Although adult literacy programs are neither new nor rare, far 
less is known about their outcomes than one would expect. 
Beyond some knowledge about the relationship between time 
invested and achievement, and the necessity for practice and 
application of acquired skills in order to retain them, we have 
very few empirical studies demonstrating the impacts of 
programs or reliable predictors of program success. 

The literature states that an array of variables have bearing on 
the success of students in adult education programs, but many 
of these variables are un- or inadequately researched. Thus 
little empirical data exists to substantiate these hypothesized 
link s, and recommendations to practitioners are, most often, 
based on guesswork and conventional wisdom rather than 
research findings. For example: 
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Literacy’s 
conventional 
wisdom is 
based on 
shared values 
and anecdotal 
information, 
not 

empirical 
examinations 
to determine 
what works. 
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The adxilt education research community generally 
considers the inadequate preparation of adult education 
teachers and volunteer instructors to be a fundamental 
weakness of adult education services. There is little research 
literature, however, that identifies appropriate recruitment and 
training strategies (Office of Policy and Planning [ED], 1992, p. 
200 ). 

The same document makes the following claim, and dtes the 
work of Kutner et al (1991) as support for the cl aim : 

Components associated with effective workplace literacy 
projects include the active involvement by project 
partners, active and ongoing involvement by employees . . 
(Ibid., p. 201). 

In fact, however, Kutner acknowledges that these components 
are based on research which is 

. . .limited to primarily descriptive studies. . . empirical 
data are not available to document the characteristics of 
effective workplace literacy programs. . . (Kutner, 
Sherman, Webb & Fisher, 1991, p. 25) 

The conventional wisdom of literacy practice derives primarily 
from shared values, anecdotal information and descriptive 
accounts of practice rather than on empirical examinations of 
hypotheses about what works. While this conventional wisdom 
often gives rise to recommendations that are sound from the 
point of view of human relations, whether these 
recommendations have any strong or predictable links to 
program outcomes is largely unknown. 

Why don’t we know morel 

Several factors explain why we don’t know more than we do 
about the impacts of adult literacy programs and factors linked 
to program success. Lerche (1985) notes that: 

. . .one of the greatest problems facing the field of adult 
education. . .[is that] literacy educators generally share 
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no common criteria for evaluating their performance. 
Standards of success range from such, limited though 
concrete definitions as the numbers of students served to 
such broad and vague definitions as making fundamental 
changes in people’s lives. . . .( p. 230). 

The lack of comparability of results of literacy programs makes 
it very difidcult for practitioners or researchers to confidently 
identify characteristics associated with effective practice in 
adiilt literacy education. 

Another reason why we know less about program outcomes than 
we might is that there has been relatively little systematic 
outcome-based evaluation of adult literacy programs. In this 
regard, adult literacy programs have much in common with 
other education programs. 

A comprehensive review of evaluation practices across Ontario s 
education system (Kindergarten through Grade 13) reveals that 
educators strongly prefer process-based evaluation over 
outcomes-based evaluation. 

Focusing on the results of teaching has been deemed to be 
impossible, impractical, or impolitic. Indeed (this) point 
of view is amply represented in the literature (Hayman 
and Sussman (1986) p. 90). 

Some reasons given for eschewing outcome-based evaluations in 
regialar primary and secondary divisions also apply to adult 
literacy education. These include: the belief that learner 
outcomes are often more influenced by factors beyond the 
control of the program than by what programs or teachers do; 
the belief that standardized measxires of student learning (often 
employed in outcomes-based evaluations) are too narrow a 
gauge of what students know and fail to measure the things we 
care about most (i.e. aspects of problem solving, the ability to 
synthesize and interpret different kinds of information, 
creativity); and the belief that instructors will start to “teach to 
the test” instead of teaching to meet the needs of the student, 
where standardized tests of learning are used as the measure of 
success. 
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Nevertheless, recent literature suggests that an outcomes (or 
results) based orientation to adult literacy program evaluation 
is increasing at this time. This represents a significant shift in 
an area where evaluation efforts, until now, have been largely 
input (or process) oriented. 

There has been more and better outcome-based evaluation of 
workplace literacy programs than of any other sort, in part, no 
doubt, because employers, acting as co-funders, have been 
insistent upon seeing documented results. As well, it is probably 
easier to specify outcomes in a work setting, where absenteeism, 
accident rate, and productivity rates are routinely measured. 

But even in this area, good evaluation is the exception, not the 
rule, if high standards are applied: 

Although federal and private support funds thousands of 
workplace literacy programs, very few programs have 
been evaluated beyond a superficial level. . . . Typical 
workplace literacy program evaluations involve anecdotal 
reports, learner satisfaction questionnaires, or pre- and 
post-restdts from a standardized basic skills test such as 
TABE or ABLE (Miktilecky & Lloyd, 1993, p. 53) 

In fact, the array of evaluative instruments, described here as 
“superficial”, extend well beyond what are employed in the 
majority of adult literacy program evaluations. 

Only a minority of literacy programs consistently measure 
learners’ levels at the beginning and end of the program, 
whether on specific competency measures or on general literacy 
skills. Many service providers are reluctant to divert scarce 
dollars away from service provision to program evaluation. 
Further, many adidt literacy practitioners are concerned that 
learners, often veterans and victims of school failure, will be put 
off by experiences that re min d them of that failure and will be 
discouraged or discomfited by anything that smacks of testing. 
There is particular reluctance to the use of standardized literacy 
tests which yield residts in terms of school grade level, using 
terms which are inevitably very discouraging to adxilt learners 
(i.e. reading at grade 5 level). As well, the strong commitment of 
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many practitioners to provide learner-centered progr ammi ng 
makes some reluctant to impose an3rthing (i.e. testing) that the 
client has not expressed interest in. 

In some cases, literacy providers simply delay assessment until 
clients are more comfortable with the literacy class or 
environment; in others, it is never scheduled, or never happens. 
The informal nature of many adult literacy programs, which 
often attempt to accommodate learners by allowing them to 
enter and to leave the program at any point during the year, 
and the high dropout rate that so many programs experience, 
means that it is very difficult for instructors or tutors to find the 
opportunity to administer assessment measures individually, or 
to know when the client is, in fact, present for the last time. 

Last but not least, few literacy providers are experienced with 
evaluation methodology. As a result, local program evaluation 
efforts are often diluted to the point that, if there are any 
outcome data, they tend to be of questionable reliability and 
validity. 

The consequences of not knowing 

The consequences of not having reliable program outcomes data 
at the local program level are that teachers and tutors are left 
with no alternative but to operate on the basis of what they feel 
they know about what works. They may ask learners what 
strategies work best for them, or feel most comfortable to them. 
They will observe, anecdotally, which sessions seem to be most 
productive, and may alter their methods based on their 
observations. And they may consult with their peers, who in 
turn will offer advice based on their own personal experiences 
and hunches. Reflection on personal experience and intuition 
are undeniably important data sources; nonetheless, 
practitioners also need empirical information about what works 
best, in order to help them make decisions about the approaches 
they will use. 



Practitioners 
need empirical 
information 
about what 
works best, in 
order to help 
them make 
decisions 
about the 
approaches 
they will use. 
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Advocates 
need outcome 
data to 
persuade 
others of the 
value of 
literacy 
programs. 

Students 
need 
objective 
information 
on outcomes 
to decide 
whether time 
invested in a 
particular 
program has 
been well 
spent. 



The lack of outcome data also makes it more difiScult to 
communicate about thp value of their efforts to fimders or the 
larger public, in a way that is convincing. While there may be 
little doubt in the public mind that hteracy is important, there is 
often considerable doubt whether particular programs are 
successful in increasing literacy, or in doing so with a 
reasonable degree of efficiency. 

Some writers also suggest a cost for the client who lacks but 
would value some evidence of her/his increased literacy beyond 
what s/he can observe. A client may know that s/he can read a 
text that formerly was not comprehensible, but may want to 
know whether that increased ability is stifficient to enable 
her/him to apply successfully for a job, get a driver’s license, or 
whatever. S/he may also want objective data about his or her 
learning in order to measure whether the time invested in a 
particular Hteracy program has been well spent, in order to 
decide whether to continue in the same program or look for a 
different one.i 

The lack of knowledge about what kinds of programs are 
successful for what kinds of clients, and why, is highly 
problematic at the macro level where delivery systems and 
related policies are developed and monitored. Do smaller classes 
make a difference to general literac3/ learners? To workplace 
Hteracy learners? Are certain materials associated with more 
rapid gains for ESL Hteracy learners or ABE learners? Do 
instructors with a particular kind of training/experience appear 
to be more successful with family Hteracy or general literacy 
groups? Are community-based Hteracy programs as effective as 
those delivered by educational or other institutions? How can 



1. It has also been suggested that: 

Lack of e.xperience with testing may underestimate the achievement 
of adiilts. When neoliterates were given testing practice, which 
children get in schools, their performance improved significantly 
(Abadzi, p. 10). 

This may or may not be equally applicable to learners who have 
attended school as children, but have not had recent testing 
experience, as to those who have never attended school, who are the 
referents in the above quote. 
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Zanders assist providers to be more effective and efficient? 

Answers to questions like these are needed to advance our 
knowledge of how to provide effective and efficient programs; 
how to best assist adult learners and best use public fu n ds. The 
need for outcomes-based «^ata is more acute at the macro level of 
analysis and planning tbart it is at the local level, while it is also 
more difficult to obtain the necessary information. Not only is 
evaluative data needed from individual progr ams , but some of it 
must be the same data so that comparisons between programs 
ran be made. In order to know whether Program A (or some 
component thereof, such as its instructional method, or 
materials) gets better resiilts than Program B with older literacy 
learners, for example, information is needed from each program 
based on the same meastire. 
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1.2 Which Outcomes? How Measured? 



Outcomes- 
^ased 
evaluation 
measures are 
usually based 
on the goals of 
Ihe program. 
What 

difference did 
the program 
Intend to 
make! Did it 
make that 
difference! 



At the micro level: evaluation of individual programs and 
student learning 

In outcomes-based evaluation, programs are usually assessed in 
terms of their goals. A family literacy program may look at 
parental attendance at school meetings, or how regularly 
parents read with their children. Such meastires are collected 
before and after, at the beginning and at the end, of the 
program, and the difference is taken to represent the effect the 
program has had on the desired ends. (Ideally there would be a 
comparison group of people who did not participate, such as 
parents on a waiting hst; but such an ideal is seldom realized.) 

A workplace program may measxire success in terms of 
decreased absenteeism, or the nximber of employee/learners who 
have applied for higher level jobs within the organization. A 
secondary school diploma eqxiivalent program can look at how 
many grade levels a learner has progressed, or how many 
credits s/he has acqxiired. Commimity programs, often organized 
on an individual basis, will look at the achievement of a 
learner’s goals: to be able to pass the driver’s test, or to fill out 
bank forms, or whatever. 

In recent years, some work has gone into describing outcome- 
based evaluation strategies that can be applied to individual 
adult literacy programs. For example, Mikulecky, Henard and 
Lloyd ( 1992 ) pubhshed a Guidebook for developing workplace 
literacy programs . In the model workplace literacy program 
they worked with, the outcomes of interest were related to 
beliefs about literacy, literacy practices at home and at work, 
literacy abilities and processes, literacy plans, and productivity 
on the job. Additional emphasis was placed on literary activities 
for learners who were parents. 

In each of these five to six areas, pre- and post-measures were 
given, with before and after results compared statistically. In 
addition, separate comparisons were made for each of three 
distinct program content groups (the report writing group, the 
promotions group, the family literacy group), so that outcome 
data would be particular to each of the three groups. Thus, they 
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were able to show, for example, that as far as changes in literacy 
beliefs were concerned, there was no c h a n ge in learners’ view of 
what makes a literate person; but there were significant positive 
changes in learners’ view of themselves as literate in two of the 
three groups. 



Measures of beliefs, practices, and plans were based on self- 
report, in which learners responded to interviewer questions 
created by the evaluators, such as “TeU me the sorts of things 
you read and write on the job/away from work during a normal 
week.” These items were created specifically for this particular 
program evaluation. 

Productivity was measured by work attendance, changes in job 
status, samples of employee written reports, and a survey of 
class participants. Except for the last, these were not self-report 
measures. Attendance, for example, was measured one month 
before class, during class, and one month after. (Such 
measurement is obviously more possible in a workplace 
program, where there is some stability in access to records over 
time.) Similarly, the report samples were collected before, 
during, and after class, for analysis. Measures like absenteeism 
are standard, and can be compared not only before and after a 
program, but across programs. 



In a cloze test, 
every fifth word 
is omitted from a 
passage of about 
150 words, and 
learners fill in 
the blanks 



The evaluators argue that workplace literacy programs, which 
differ significantly from general basic skills programs, need to 
measure the types of literacy gains that result from specific 
workplace-tailored programs. They devised a set of questions 
based on materials used on the job (which were also the 
materials on which instruction was based), so that learners had 
to show their comprehension of the text by finding the answers 
within it. (Learners were told what page to look at, and had to 
extract the answer from what they foxmd on the page.) Some 
questions were of the multiple choice variety, others had only 
one correct answer. The evaluators also used the same 
materials to construct a cloze test, where every fifth word is 
omitted from a passage of about 150 words, and learners fill in 
the blanks. This test was also based on texts the learners 
actually use on the job. 
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Good 

outcome- 

based 

evaluations 
include 
measurement 
of objectives 
like change in 
beliefs, 

attitudes and 
self-image, or 
in work- or 
home-related 
literacy 
practices. 

They also 
include the 
measurement 
of literacy 
gains. 



This evaluation scheme includes a variety of sources of data. 
Self-report data, appropriately collected in interviews before and 
after the intervention, is balanced by objective data on 
productivity and literacy gain. Such an evaluation scheme 
would teU learners and program providers what degree of 
gain/success was associated with their efforts/activities. But, 
except perhaps for absenteeism, the measures are non-standard, 
and it would be very difficult for anyone to decide whether other 
workplace literacy programs being offered were more or less 
successful than these, unless they deliberately adopted the same 
measures. Among the three programs under scrutiny in this one 
study, however, such comparisons could be made, because the 
measures were common ones. And any other group of providers 
of workplace literacy programs, for example all those within a 
particular geographic region, could jointly design measures in 
common. Thus, a cloze test of the same length, and scored in the 
same way, based on texts at the same level of difficulty, could be 
used by different programs, even if the exact text used in each 
program differed, based on the particular materials used in each 
program. 

Good program evaluation models, such as the workplace model 
developed by Mikulecky and others and described above, are not 
narrow in focus, and they do not exclude measurement of 
objectives like change in beliefs, attitudes and self-image, or in 
work- or home-related literacy practices. But neither do they 
omit the measurement of literacy gains, whether standardized 
general literacy tests or purpose-designed context-based tests 
are the measure. 

Another issue in evaluation is whether the same measures 
should be used to assess both individual progress and program 
effectiveness. It is certainly possible and efficient to make one 
measure or set of measxires serve two purposes, as when 
individual gain scores are examined as a measure of how much 
benefit was derived from the course by the learners. On the 
other hand, there are advantages to using different tools for 
individual assessment and program evaluation, despite the 
extra work involved. First, clients can be reassured that the 
program evaluation measxires are not about individual 
achievement or ability, and are thus likely to feel less anxious 
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and to be more successful (see the work of Steele). Second, the 
goals of the program may be several, in addition to general 
literacy (such as reduced absenteeism, increased participation in 
school meetings, etc.), and as such pertain to the program, not 
the individual. As one writer clearly states. 

Many of the problems with standardized testing 
experienced by programs are due to the attempt to use 
one test for both program accoimtability and instructional 
decision m akin g. For instance, using the Test of Adult 
Basic Education (TABE) for pre and post-testing to report 
gains in general literacy to state and federal 
administrators is a program accoimtability function of the 
tests. 

But using the TABE to assess learning in a specific 
literacy program, in which learners may choose to read 
and study a technical manual, is an inappropriate use of 
the test for assessing either instructional needs or 
progress. In this case, an alternative assessment method 
is needed, perhaps one in which learners’ needs are 
determined by inter'/iews that include trial readings of 
technical manual passages. Then, progress checks using 
reading aloud and question/discussion periods for 
checking comprehension might be used to indicate 
learning in the program. (Sticht, 1990, p. 28) 

With many standardized literacy measures, test scores are 
interpreted as grade level equivalents, and this makes them 
discouraging to learners. If such measures are used for program 
evaluation only, and not reported individually, such 
discouragement is avoided, without sacrificing standardized 
assessment of literacy for reasons of program assessment. 

Learners need to develop specific, workable instructional 
goals so that there can be immediate results. Being able to say “I 
can read these signs now” or “I’ll be able to read all these 
directions in a few weeks” is far better for morale than having to 
wait a year or more to be able to say, “Now I can read at the 
(fifth)-grade level. Most adult learners do not want to express 
their learning goals in terms of elementary school grade levels, 

11 

A Review of Issues and Literature Related to Literacy Program Evaluation 
Ontario Literacy Coalition, 1996 




19 



Measures that 
Jan he applied 
icross a 
n.umher of 
Drograms 
make it 
Dossihle to 
Identify the 
system-wide 
changes likely 
to result in 
Improved 
adult literacy 
practices. 



and instructors should honor their preferences. (Lord, 1994, p. 
16). 

At the macro level: advancing our knowledge of what 
works 

Presumably all literacy programs have in common the goal of 
increasing their students’ ability to read and write (and 
sometimes to do basic mathematics), almost always in the 
language of the majority. (There are some ESL programs whose 
goal is to help learners become literate in their own mother 
tongue, usually as a stage on the way to English literacy.) 

Hence, it is suggested that all programs need to measure the 
extent to which each of their clients improves her/his reading 
and writing (and perhaps also listening and speaking) skills 
while enrolled in the program. Such assessment would be 
necessary at entry, as a diagnostic on which to plan the learner’s 
program; it might be necessary at some set intervals to check on 
progress and comprehension; and it would be necessary at exit 
to inform the program providers about the effectiveness of their 
procedures. 

General measures that can be applied across a nximber of 
programs are needed to help planners and policy makers 
identify what system-wide changes are likely to result in 
improved adult literacy efforts. If existing measures, such as 
those developed for the school-age population, or those already 
created especially for adult learners^ are truly unsatisfactory, 
then new standard measures (such as the cloze test, with 
specific marking and reporting procedures) can and should be 
developed. 

The recently published National Evaluation of Adult Education 
Programs (NEAEP, 1992 - 1994) was able to compare hundreds 
of programs because each one of them reported that it used one 
of two standard tests of general literacy designed for adults. 
Such comparisons enabled the evaluators to conclude that 



2 A very useful discussion of the nature and value of standardized tests of 
adult literacy is found in Sticht, T. G. (1990). 
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having full-time staff and individualized programming meant 
greater literacy gains for ABE learners, and that higher per 
client costs were associated with higher gains for ESL learners, 
giving program providers and funding agencies some evidence of 
the value of their investment. But, as the NEAEP reports, their 
sample was far smaller than it sho^lld have been, and as a result 
the literacy test gains, while they were in the expected direction, 
were not as helpful or conclusive as they had been expected to 
be. 



When the study was designed, the plan was to use the 
results of the analyses of reading achievement tests as 
the basis for this analysis (of the number of clients who 
were successfully served; that is, received sufficient 
literacy instruction to significantly reduce their literacy 
needs). However, programs were generally unable to 
provide valid pre- and post-test scores on clients 
participating in the study, and analyses of the relatively 
small number of achievement test scores do not support 
their use as indicators of literacy- needs reduction. 
Consequently, the time required for clients to move from 
one instructional level to the next was used as an 
alternative (Young et al, 1994, p. 41). 

In fact, although there were about 18,000 learners represented 
in the programs surveyed, there was complete test data (pre- 
and post-) on only just over 600 of them. While instructional 
levels achieved could be a perfectly adequate measure if 
standardized, without standardization they do not provide a 
means of comparison. Is intermediate ABE the same in 
Program A as in Program B? What about advanced ESL 
Literacy? (In the NEAEP programs some of these levels were 
standardized, but some were not.) 

The NEAEP data include very few community based programs. 
One smaller scale review, in the late 80s, looked at the outcomes 
of nine community based programs which volunteered to be 
evaluated. Ten criteria were chosen, but not every program 
chose to be evaluated on every criterion. Six looked at reading 
res^llts, five at writing, and two at math. Three of the six 
showed gains in reading, two of them large gains; one of the five 
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showed gains in writing; and one of the two showed math gains. 
Two programs showed significant gains in commiinity activities 
and contributions made by learners; seven of nine showed 
measurable gains in self esteem; and three of seven showed 
significant gains in “self determination” (self direction, 
planning, persistence, etc.). 

A reviewer of this data concludes that: 

. . .on the standard measures of impact, such as reading 
and mathematics gain, community based programs 
compare favorably with traditional ABE programs. 

(Beder, 1991, p. 118) 

Of course, other goals in addition to general or contextualized 
Hteracy can (and probably should) also be measured in 
comparable ways across programs. Programs which aim to affect 
family/in tergenerational literacy, for example, can ask questions 
like “In the last 7 days how many times have you read/looked at 
books with your child or listened to him/her read?” “In the last 7 
days how many times have you helped your child with 
homework and/or with school projects?” (See Mikulecky & 

Lloyd, 1993a.) The answer is niimerical, on a scale of 0 to 10+. If 
such questions are asked at the beginning and end of programs, 
comparisons can be made across programs which can tell us 
something about what program components most affect family 
literacy. 

But whatever measures are used to evaluate programs, it is 
probably true that without some measure of literacy gain which 
is not based on self-report, program evaluation is incomplete 
and inadequate. 

. . .while . . .alternative methods [of assessment] are 
invaluable for their contributions to learner progress, 
there are limitations to the exclusive use of such 
techniques for learner and program evaluation . . . 

One of the problems identified by alternative assessment 
providers is the fact that, although standardized, 
nationally normed tests fail to match program content, 
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administrators, teachers, and millions of other adults can 
and do perform very well on any or all of the dozens of 
standardized tests of reading, writing, and arithmetic. . . . 
The question is raised, therefore, of whether or not adult 
learners in ABE and ESL programs are being directed to 
less demanding levels of achievement if they are not 
evaluated using standardized tests. (Sticht, 1990, p. 27) 

An example of an evaluative comparison of three groups of adult 
literacy programs carried out in the absence of objective and 
standardized data is the Review of the Basic Skills, Co m mu m ty 
Literacy, and Basic Skills in the Workplace Programs (Woods 
Gordon, 1990). The programs were evaluated on the basis of 
interviews with and questionnaires to program providers, and a 
telephone survey of previous and current clients. Most of the 
report addresses inputs or process, rather than outcomes. 
Highlighting client and program characteristics, the report is 
useful in describing what goes on in the programs in some 
detail. In the section which addresses program effectiveness, 
there are discussions about varying perspectives and conceptual 
approaches, but there is in fact no data on literacy achievement, 
because no measure of literacy gains was employed. 

Nonetheless, apparently based on the fact that “Colleges spend 
considerable time on literacy instruction” (p. 76) and “Research 
has shown that. . . a significant amoimt of practice time actually 
doing reading and mathematics is required for gain to occur ’ (p. 
75), the evaluators conclude that: 

The OBS [Ontario Basic Skills] programs are, in general 
increasing the basic skills of the workforce . . . (p. 89) 

It is also reported that “Community groups [OCL programs) 
have trouble delivering sustained, long term training” (p. 77). 
This seems to imply they are necessarily less successful 
outcome-wise. Yet, in the learner survey, 47% of OCL 
participants said they felt they can read better as a result of 
their program, compared to only 13% of the OBS participants. 
All of this suggests that, in the absence of objective data on 
literacy achievement/gain, evaluators have difficulty arriving at 
vahd conclusions for which they can offer solid evidence. 
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Adiilt literacy programs exist to help adults who are less t han 
sufficiently literate to become so. Successful programs, 
whatever else they may do, are those which accomplish this end 
for an appreciable percentage of their clientele. Literacy 
providers anH learners have much to gain firom the application 
of well thought-out and well-executed outcomes-based program 
evaluations, when evaluations are coupled with a real 
commitment to make effective programs available to learners. 
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2.1 Introduction 



Despite the need to implement improved evaluation 
procedures and generate more substantial effectiveness 
data, there is much that we do know about what works, 
and we have identified successful practices and programs. 
(Lerche, Ibid., p. ) 

In this section we review research on the connections between 
characteristics of literacy programs and program outcomes. By 
far, the most comprehensive and ambitioiis attempt to describe 
the relationship between adult literacy programs and their 
results in terms of literacy acquisition, and to tie program 
characteristics to student gains, is the recently published 
National Evaluation of Adult Education Programs (NEAEP , 

1992 - 1994). All U.S. government funded adult education 
service providers were surveyed, and data obtained from over 
2600 programs covering more than 20,000 clients. Most of the 
programs were administered by local education agencies (69% of 
programs, 73% of clients) and community colleges (17% of 
programs, 18% of clients). Programs were divided into three 
types: ABE (Adult basic education), ASE (Adult secondary 
education) and ESL (English as a second language). Client and 
program characteristics were documented, as were test-based 
gains in basic skills, and other results (based on follow-up 
interviews). The NEAEP study gives us some indications about 
which features of literacy programs make the most difference for 
these three kinds of clients. 

The study examined many of the issues with which North 
American adult literacy educators and researchers have been 
most concerned (e.g., factors correlated with retaining students 
in programs; the relationship between retention and 
acMevement; correlation between instructor qualifications, 
modes of instruction, and achievement). The NEAEP study will 
be a main point of reference in this section of the repoi't, as we 
consider the following issues: retention of students in programs; 
the amount of time it takes for adults to acqxiire or improve 
their literacy skills; the correlation between other program 
variables with learning outcomes; and the correlation between 
client factors and learning outcomes. 
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The adiilt education literature places enormous emphasis on 
student retention and attendance (that is, the length of time 
students stick with a program, and how often they attend). This 
is because (1) retention and attendance rates tend to be very 
low; and (2) amount of instruction is believed to be directly and 
closely related to literacy acquisition. In the absence of data on 
actual outcomes in terms of learner gains, retention data have 
become a substitute for achievement data. It is assximed that the 
longer a student stays in a program, the greater the learning 
gains and the more effective the program. For this reason, we 
will look at data on retention in addition to that on 
achievement. 

Very poor attendance and retention rates characterize adult 
literacy programs. T}rpical dropout estimates in North America 
and elsewhere are around 50%, plus or minus 20%. An Ontario 
study found a 38% attrition rate among enrollees in basic skills 
programs (Woods Gordon, 1989, p. xii). The NEAEP survey 
found that 15% of enrollees drop out before their course begins; 
almost one third of enrollees are gone before the end of the first 
month; and 44 per cent who begin leave within two months. 
Eleven per cent of all new enrollees continue into a second year 
of instruction. This suggests that the program offered during the 
first month is crucial for long term persistence. A similar figure 
is reported for non-attendance, described as a “harbinger of 
dropout”, from as far away as Kenya, where the literacy 
programs with the best records of completion and achievement 
(urban) showed a 30% absenteeism rate, and those with the 
poorest (rural) a 50% absenteeism rate. (See Abadzi, 1994, p. 8.) 

Why are dropout rates so high? It is clear that there are several 
reasons for the high dropout rate (as well as a high absenteeism 
rate) among enrollees. One reason suggested by research is 
students’ frustration about the “enormous amount of time 
required to learn to read and about feelings of failure and 
depression during learning plateaus” (Chall, Heron, and 
Hilferty, 1987, p. 191). Learning ability does not decrease with 
age, but more time is needed. (See Balmuth, 1986, p. 36.). 
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A literacy acquisition rate which is natural and comfortable and 
socially acceptable for a child can feel like a snaifs pace to an 
advdt, who is trying to catch up with more literate peers as 
qthckly as possible. This is particularly true, apparently, for 
advdts who have a histoiy of poor prior school performance, and 
who may have learning disabilities which substantially increase 
the amount of time needed to acquire literacy skills. 

... it was frustra tin g' to attend school with the realization 
that, with the rate of progress they were making, it wovdd 
take years and years to reach a point where their 
education would make any difference in the money they 
earned and in the way they lived. (Balmuth, p. 38) 

Research on how much time it takes for adults to learn to read is 
examined more closely later in this report. 

There is growing general agreement in the literature that with 
respect to retention of students, aspects of the programs are as 
important as are characteristics of the learners, as well as being 
much more amenable to change. This is a shift from earlier 
literature which tended to describe only learner and situational 
characteristics (poverty, lack of prior success in education, 
shifting employment opportunities), not program characteristics, 
in attempting to explain student attrition. 

Essentially, most of the factors that were fotmd in the NEAEP 
study to be substantial predictors of persistence are factors over 
which local programs have some control. For clients who had 
already attended at least 12 hours of adult education 
instruction, the explanation for sustained attendance lies 
primarily in factors related to program structure and design, 
rather than personal background or motivational factors. 

According to the NEAEP evaluators: 

The strongest predictors of persistence across all . . . 
components [i.e. ABE, ESL, ASE] are as follows: the 
presence of support services that clients actually use, the 
receipt of instruction during the day as opposed to 
evening hours, and the type of learning environment in 
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which the clients participate. . .(Development Associates, 
1994, p. 47) 

Recommendations for literacy programming often include the 
provision of ancillary services, especially counseling, and the 
integration of such services, to improve client access. (See 
Balmuth, 1986; Lerche, 1985.) The NEAEP study indicates that 
about one quarter of clients used one or more of the support 
services provided by their program, counseling being the most 
commonly used. Those who used such services were more likely 
to remain in the programs longer; put differently, programs 
which provide useful services have clients who persist longer. 

Clients in programs that provided five types of services or more 
received, on average, 115 hours (and 19 weeks) of instruction, 
whereas clients in programs that provided four types of support 
services or less received, on average, 60 hours (and 17 weeks) of 
instruction (Young, Fitzgerald & Morgan, 1994, p.p. 23/24.) The 
authors hypothesize that the availability of services such as 
counseling, financial assistance, transportation, child care, case 
management and other services may explain why some clients 
can sustain participation while others cannot. 

The fact that day time rather than evening instruction is related 
to persistence is probably because day time attendance is 
associated with longer hours (evenings are shorter than days) 
and because time of attendance is related to employment or 
family characteristics. 

Type of learning environment (classroom or lab or both), with or 
without a component of independent study, also makes a 
difference. For ESL clients, the inclusion of an independent 
study component or participation in a learning lab, in addition 
to classroom learning, predicts longer retention than a 
classroom only program. For ASE clients, the lab is important, 
but not the independent study. But for ABE clients, the 
contrary is the case: retention is longer for those in classroom 
only programs, which are more structured and possibly more 
nurturing for learners at the most basic levels. 
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But perhaps what is most important about the NEAEP findings 
on retention and persistence is that they do not provide 
lonqualified support for what is the constant assxnnption in the 
literature, which is that whatever increases retention improves 
achievement. While it may be useful to docu m e n t that the 
provision of support services improves retention, it does not 
appear that improved retention is a consistent predictor of 
literacy acquisition. 

Some reviewers suggest that a high dropout rate should not 
necessarily be viewed as reason to substantially alter a 
program. Based on observations of literacy progr amm i n g in 
developing countries, the authors conclude that “A 50 percent 
dropout rate is acceptable and does not necessarily indicate that 
materials and instructional design need improvement” (Abadzi, 

p. 10). 
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Age is not an absolute barrier to the acquisition of literacy. 
Substantial improvement has been foxmd to occur at all ages, 
and “age-related differences in cognitive abilities probably do 
not contribute to mearungful differences in either the efficiency 
or effectiveness of learning to read.” (See Abadzi, p. 75) Within 
the adult popxilation, age-related differences in hteracy learning 
are apparently small, especially when contrasted with the 
differences within childhood (as between ages 6 and 18). 

Adxilt learners are as adept at acquiring literacy as children, but 
it takes longer (see Balmuth, 1986, p. 36). It is equally true, 
however, that adult literacy learners actually progress faster per 
week/month of instruction than yoimg children, possibly 
because the programs are more focused and concentrated than 
those presented in elementary school. Thus, while it may take 
an adult about 120 hours of instruction and practice to gain the 
equivalent of one year in measured reading level, the same one 
year gain represents many hundreds of hours over a school year 
for the average school child. 

In a review of the evidence on age-related aspects of literacy 
learning, Abadzi (1994) cites studies which suggest that while 
adults acquire basic literacy (decoding, alphabetic skills) faster 
than children, they may acquire fluency more slowly, 
necessitating considerably more work and practice to move from 
simple decoding to a level of fluency that encourages use. Oral 
fluency in a second language may be acquired by adxilts much 
more easily than written fluency. (Much of this evidence is 
based on the experience of second language literacy learners, 
but may apply to first language literacy learners as well.) 

Because it has seemed self evident that amount of instruction is 
related to Hteracy gain (Balmuth, 1986, p. 31), the capacity of a 
program to retain students and to maintain high attendance has 
been ass um ed to be related to its effectiveness in increasing 
Hteracy learning. The operative assximptions have been: the 
more hours the better; and, the longer the time period (the more 
weeks, months, years) the better. This latter rule is based on 
findings in psychology which indicate that it is better to 
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distribute learning (no matter what is to be learned) over time, 
rather than “mass” it together intensively. But the information 
available to substantiate the relationship and give it some 
exactitude has been largely lacking. 

Therefore, although the figure of 12 hours as a minimum 
amoTint of instruction for progress toward literacy has been a 
criterion of federally funded programs in the United States, the 
NEAEP study finds essentially no systematic differences 
between learners with less than and more than 12 hours of 
instruction, suggesting that funding criteria may have little 
relevance to actual results or program potential. 

Recent findings are filling in the essential details of how much 
instruction is enough, and for whom. 

No program . . . has been able to consistently improve 
reading ability from low level to high school or college 
standards in 20, 30, or even 50 hours (Mikulecky and 
Lloyd, 1993, p. 5). 

Thus, for low literate adults, it seems advisable to provide 
several himdred hours of practice with materials within 
the reach of the learner to insure that. . . transfer is 
obtained (Mikulecky, Albers, and Peers, 1994, p. 21). 

The figure of about 100 to 120 hours is often cited as a typical 
time requirement for a one year gain in reading skills. 

One fairly large scale study reported that when 

...several hundred learners in a prework literacy program 
in 13 diverse sites . . . were tested for reading gains using 
the Test of Adult Basic Education (TABE), an average .7 
of a year gain in reading ability after approximately 100 
hours of instruction was demonstrated. (See Mikulecky 
and Lloyd, 1993, p. 4) 

Another review concludes that: 
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A significant amount of practice time actually doing 
reading and mathematics is required for gain to occur. 

[An average of about 100 hours per grade gain.] (Woods 
Gordon, 1989, p. 75) 

The evidence from developing coxintries appears to suggest an 
even longer time period is necessary for the estabfishment of 
basic hteracy among adults who lack all Hteracy skills; Abadzi 
(1994, p. 10), reviewing that Hterature, says that: 

Six month classes of 200 hoxirs or less are often preferred 
for their lower costs but have high failTire rates. Some 
studies suggest that the time allotment may be cut by as 
much as half in programs which are highly 
contextualized; where the goals, materials, and 
instructional methods are very close to the context in 
which clients will be applying their skills, as in many 
workplace literacy programs. (See Mikulecky & Lloyd, 
1993, p. 5) 

When pains are taken to test these various assumptions, the 
findings are less than unidimensional and simple. The NEAEP 
study found, as have several others, that ESL clients tend to 
complete more hours of instruction than ABE or ASE clients. In 
that study, the ESL clients received a mean of 120 hours of 
instruction, compared to 84 hours for ABE clients, and 63 hours 
for ASE clients. In the NEAEP study, a different test was used 
to meastixe achievement for ESL learners than for ABE and 
ASE learners. Hence it is not possible to compare ESL to 
ABE/ ASE gains, in order to determine whether the longer 
n umb er of mean hours of instruction of the ESL group is 
associated with larger learning gains, at the group level. 

It is possible, however, to compare the ABE to the ASE group, 
because both groups were tested with the TABE at entry and 
exit from the programs. In fact, the ABE group, with its average 
of 84 hours of instruction showed an average gain of 15 TABE 
scale score points, a gain of 1.3 years in grade equivalency, 
compared to an average gain of 7 TABE scale score points for 
the ASE group, a gain of .8 years in grade equivalency, which 
averaged only 63 hours of instruction, suggesting that the 
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longer instructional period was associated with enhanced 
learning. 

Another way of assessing how long it takes to acquire needed 
skills is to document the average hours required for clients to 
move from one to the next level of a course offered within an 
ESL or ABE program. The NEAEP found that the median hours 
needed to complete the beginning level ABE course was 32; for 
the intermediate level, 36 hours were needed. The time 
requirements for ESL were vastly larger; median hours to 
complete beginning ESL was 216, and for intermediate level 
ESL, 136. 

But the NEAEP findings concerning hours of instruction and 
achievement are surprising also. When the relationship between 
hours of instruction and achievement gains was exarmned 
directly, it was found that the relationship was statistically 
significant only for the ESL group. That is not to say that all 
three groups did not make literacy gains; they did. But only in 
the ESL group could length of instruction time be said to have a 
direct effect on learning gains. For the ABE and ASE groups, 
other program characteristics were related to individual 
achievement, but hours/attendance was not. 

While it may seem self evident that more instruction will result 
in more learning, the fact that a carefully designed, large scale 
study could document that relationship for only one of the three 
component groups (i.e. ESL) suggests that it is not reasonable 
for policy makers or program providers to assume that simply 
retaining clients for more hours will result in greater learning 
gains. Such gains need direct measurement. It may be perfectly 
true that many hours of instruction are essential to adult 
acqmsition and improvement of basic skills; but it may be 
equally true that increasing hours of instruction is not sufficient 
to assure increased knowledge. 

Furthermore, the issue of how much time is required to derive 
benefit from programs is complicated by the finding that newly 
acquired literacy skills are fragile; they are easily lost. 
Individuals who have completed literacy training programs may 
not retain their skills very long or at a level high enough to be 
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useful. Just as considerable time must be invested in practice to 
acqxoire literacy, so must practice continue after literacy 
acqmsition, for maintenance. 

Second language literacy and the issue of learning time 

The NEAEP finds that ESL clients are both the largest and the 
most rapidly growing component of the adult education market. 
There are indications that this is also the case in Canada, 
especially in Ontario. But the ESL population is described 
monolithically. We apparently lack any solid data which speak 
to the presumably differii^ instructional needs of those who are 
Hterate in their first language, as opposed to those who are not. 

There is reason to assume that the latter group will need more 
time to acqmre literacy in English, but how much more time is 
simply not known. Conversely, one assumes that ESL learners 
who are literate in their first language will be able to acquire 
English literacy more qmckly than English speaking advdts who 
have not acqmred literacy; but, again, there are no data which 
explore this relationship within the context of adult education. 
Most writers appear to assume that the majority of ESL 
students are not literate in their mother tongue, but no numbers 
are provided. 
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2.4 Program emphasis as a factor 

While all adiilt literacy programs are trying to develop some of 
the same generic skills, programs differ in context and content. 
In addition to general basic skills/ESL programs, there are 
workplace literacy progr am s, family literacy programs, life skills 
programs, and others, each of which has a particular context, 
and materials appropriate to, indeed often directly derived from, 
that context. 

Programs can also be contextualized by geography and clientele: 
for example, one might offer an ESL literacy program to Chinese 
speakers in a particular neighborhood, emphasizing local 
services and their access. In addition, programs can be 
organized around a variety of content areas, ranging from 
general learning programs emphasizing reading and writing in 
much the same way as is done in elementary school^, to those 
which are highly contextualized, such as many workplace and 
family literacy programs. In the latter case, learners are 
presumably very directed in their motivation and expectation, 
and the materials are similarly focused. 

In workplace literacy programs, the materials may be chosen 
because they are the same material the workers need to be able 



^ On the issue of whether the same methods should be used with adult and 
juvenile literacy, the following is instructive: 

Research with adults learning to read shows that adult literacy students are 
more likely to approach reading through bottom up approaches [where 
comprehension is built up by accumulating smaller pieces, sound by sound 
and word by word] . . . ; when reading is viewed as a process of creating 
meaning, however, students are more likely to progress and learn to read at 
a faster rate . . . Many adults in literacy programs failed to learn to read 
using subskill approaches in the public schools, so it is possible that illiterate 
adults are a kind of selected sample. . . . Perhaps the most competing 
argument for the interactive approach [combining bottom up methods with a 
top down approach which emphasizes the use of prior experience^ with 
concepts to construct the meaning of the text] comes from case studies in 
which instructors document their success using this approach with students 
with whom they had been unsuccessful with traditional bottom up 
approaches . . . (Fingeret, 1984, p. 13) 
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to read on the job. In family literacy, the content may be books 
parents would like to be able to read to their children, notices 
which the school sends home, school registration forms, and the 
like. One advantage of such contextualized learning is, 
presiimably, that its ptirpose is very clear to the learners; and so 
is its achievement. One knows when one has mastered a 
particular mamifll or book; and if the mastery of that content 
piece is directly relevant to one’s goal, its value is obvious. There 
are also advantages for program planners: 

Making principled decisions about program goals and 
implementation is often easier for programs receiving 
money targeted for a specific effort such as family literacy 
or ESL for the homeless than it is for those that are 
merely adding an additional literacy class to an existing 
program, because targeted efforts allow programs to begin 
with a clearly articulated philosophy of literacy teaching. 
(Wrigley, 1993, p. 2) 

Furthermore, the practicality of the program is appealing to 
funders, such as employers and school districts. 

The transfer of literacy skills between different contexts 

Research on literacy transfer, that is the ability to transfer 
literacy skills from one context to another, suggests that such 
transfer is not easily made, beyond the most basic operations 
(such as eye movement across the page). 

. . .most studies reveal a relatively low degree of 
correlation between reading performance with different 
sorts of material requiring differing background 
knowledge and reading strategies. . . . Learning. . . 
mindful, high road strategies (i.e. su mm arizing, problem 
solving, studying, writing for multiple audiences, editing, 
etc.) does not appear to transfer. . . automatically. 
(MikvJecky et al, 1994, p. 21) 

This finding that literacy transfer across different contexts is 
very weak is well supported by research, and has led an 
increasing nximber of funding agencies and providers to develop 
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contextualized programs, especially workplace and family 
literacy programs. 

Contextualized learning thus has the advantage of being 
directly applicable to the desired skills (transfer is not a 
problem), as well as being intrinsically motivating because of its 
direct relevance to the learner’s situation as well as the 
transparency of individual progress. There may also be extrinsic 
motivation built in, in the form of a promise of a job or of 
promotion, or at least a recommendation from a supervisor. 

Such factors presumably result in higher attendance and 
retention rates, and more learning. 



Program 
providers need 
to have clear 
goals for what 
they want to 
achieve in the 
limited time 
that learners 
are with them. 



While earlier studies of ABE and ESL students suggested that 
learners’ motivation was directed more toward general self 
improvement than to the acquisition of specific job skills (viz., 
Beder, p.p. 58-65), more recent data suggest that greater 
efficiency and effectiveness are associated with contextualized 
programs. In fact, even the earlier studies to which Beder refers, 
and which indicated that job advancement was not a primary 
goal of students in general ABE/BSL programs, found that 
improved ability to carry out family responsibilities (such as 
assisting children with their school work) was indeed very high 
among learners’ goals. 

The strength of contextualized programs can also be seen as 
their weakness: because the material and work are targeted at a 
particular domain/context, learners’ progress is specific to the 
material learned; it does not transfer very well to more 
generalized literacy gain. The conclusion of a carefully designed 
study of three workplace literacy programs was that: 

The results of this study indicate what can be expected of 
effective workplace literacy programs. Instruction has 
produced some improvement in all of the areas assessed, 
but gains appear to be limited to areas directly addressed 
in class. There is apparently no transfer of learning into 
areas not covered by instruction. Because of this, it 
appears that program providers need to have clear goals 
for what they want to achieve in the limited time that 
learners are in class. They should also seek ways to 
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extend this time beyond the classroom. (Mikulecky & 
Lloyd, 1993, p. ). 

Thus it is argued that the appropriate measiire of a workplace 
literacy program, for example, is a test or tests based on the 
materials which were used in the program, not a general 
literacy test, on which the learners are likely to show very 
modest gains, because of lack of transfer from the specific to the 
general. Clearly the issue is one of purpose: if what is wanted is 
an increase in general literacy skills for school completion, or for 
general personal upgrading, then contextualized programs are 
not the answer. On the other hand, given the problem of 
attracting and keeping clients in literacy programs, the practical 
value and motivational aspects of contextualized programs 
make them a very attractive and usefixl alternative. As well, 
they appear to have a significant advantage in that their direct 
application to the learners’ daily life means that the opportunity 
to extend practice time (through out-of-classroom use) can make 
a critical difference in the amount of learning that occurs in a 
given time period. 

Workplace Literacy Programs 

Workplace literacy programs, like any other literacy programs, 
cater to a variety of learners, including those with no literacy 
skills, limited literacy skills, and those who lack oral English 
(ESL learners). Increasingly, workplace literacy programs are 
becoming multi-stranded, to cater to that diverse clientele, 
including ESL classes, classes for those pursuing high school 
equivalency, basic skills technical coiirses, etc. But what all 
strands have in co mm on is the integration of workplace basic 
skills instruction, and the use of materials encountered in the 
work setting. 

A very significant advantage for targeted programs using 
materials learners encounter in their everyday work activities is 
that the amount of time needed for improvement may be 
decreased by half, to about 50 to 60 hours per grade level 
instead of over 100. One study found that when military 
enlisted men taking literacy classes were divided into two 
groups, the group receiving 120 hours of general instruction 
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averaged an improvement of .7 grade levels in reading ability, 
while those trained with workplace materials in the same 
amount of time improved 2.1 grade levels when reading work 
related materials Tnot necessarily when reading other 
materials). (See Mikulecky and Lloyd, 1993, p.p. 4/5.) While 
there is limited transfer between workplace literacy training 
and general literacy, it is equally true in reverse. When the 
intention of workplace programs is to increase job skills, it 
makes sense to use workplace materials to increase workplace 
literacy. 

Furthermore, targeted, contextualized programs decrease 
literacy loss over time. New knowledge must be used, or it is 
lost. Thus, one study foimd that 80% of the gains acquired in 
general literacy classes were lost within eight weeks; the 
exception was when job related materials were used to teach 
literacy; then gains were maintained, presumably because 
learners continued to use/practice what they had learned {Ibid., 
p. 7). 

An impressive piece of research on the effects of a workplace 
literacy program followed over 200 basic skills trainees in the 
banking industry for 15 years, comparing them to entry level 
employees with adequate (i.e., high school level) literacy skills. 
After several months of formal training combined with on the 
job, supervised clerical experience and counseling, the trainees 
caught up to the employees in literacy skills. Two thirds of them 
were placed in jobs, and the majority earned as much as their 
entry level peers, who had had more education and experience. 
The trainees also stayed longer on the job, on average, although 
other opportunities were available to them (Hargroves, 1989, 
reviewed in Mikulecky and Lloyd, 1993, p.p. 9/10). This program 
combined study and practical work, and had a counseling 
component. As well, and characteristic of most workplace 
programs identified as successful, it included active involvement 
of workplace partners. This kind of evaluation, based on 
measured literacy skills in addition to fimctional or competency 
based measures, such as the ability to qualify for or obtain 
appropriate employment, is becoming more common, although 
longitudinal data of this magnitude is very \musual. 
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Family Literacy Programs 



In their review of workplace literacy programs, Mikulecky & 
Lloyd (1993, p.p. 34-40) point out that some programs have 
deliberately or serendipitously had effects on family literacy: on 
the ability of parents to serve as literacy role models for their 
children. Sometimes this is a major motivation of learners, even 
if it is not the primary goal of the employer or the program 
designers. 

Similarly, Beder (1991, p.p. 112-113), reviewing four studies of 
the impact of general adxilt literacy classes on clients’ families, 
concludes that “adult literacy education does impact on the 
family in a positive way”, and gives examples, based on self- 
report, concerning help with homework and talking to children 
about school. 

While there are programs, often run by schools and school 
boards, whose content and materials are targeted to the context 
of school, and whose goal is to increase children’s literacy/ 
potential by increasing parental skills and role modeling 
abilities for their school-age children, there is almost no 
evaluative research on the effectiveness of these programs. 
There are also programs which attempt to increase literacy of 
parents with preschool children, which often use appropriate 
story books as their texts. One such program which was 
evaluated appeared to have an effect on the children’s behavior: 



they attended more closely to stories and to parental instruction 
ilhid., p. 39). 



An example of a successful intervention made with women most 
of whom had at least a grade nine education, is an 
intergene rational literacy program which aimed to positively 
affect children’s prospects by increasing their mothers’ skills 
and self confidence. Between 1988 and 1990, a study was made 
of nine such co mm unity-based programs. Based on self report 
data by more than 450 clients in these nine sites, and on case 
studies which included interviews at four of the sites with 48 
mothers and with their children’s teachers, the researchers 
concluded that, in addition to substantial literacy gains for the 
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mothers, 65% of the children demonstrated educational 
improvements as a resiilt of their mothers’ participation in the 
program. Teachers reported attitudinal and academic 
improvements, in many cases. Mothers reported that they more 
often read to their children, took them to the library, talked with 
them about school, talked with their teachers, helped with 
school activities and attended school activities. The researchers 
conclude that: 

Findings from the. . . project challenge the notion that 
adult education programs intervene too late to solve the 
nation’s education and skill problems. They call for a re- 
examination and re-investment in the education and 
training of adult women— especially those with children 
(Van Fossen & Sticht, 1991, p. v). 

While this evaluation is based principally on self-report, which 
is always subject to inflation of positive responses, it is 
nonetheless a positive statement about the possible link 
between mother-child literacy in an intergenerational program. 
The duration of the programs varied, but tended to be relatively 
brief and of only moderate intensity. 

Family literacy programs in the context of school, using 
children’s books and texts, report cards, school notices, and the 
like, would appear to have great potential for both adult and 
child literacy. The likelihood of high motivation is great; so is 
the natural opportunity to practice the skills being acquired in 
the home setting. The need for evaluative research is acute. 
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2.5 Other Factors Correlating With Outcomes 



The NEAEP study examines several relationship^ between 
program outcomes and other variables and tells us that these 
relationships are complicated. What is important for ABE 
learners is not necessarily critical for ESL learners, and vice 
versa. As for the characteristics that can and do vary among 
programs, and which are often expected to make a difference, 
the difference they make depends on who the clientele is, 
according to the following NEAEP findings: 

Staff 

• full time staff (administrative and instructional) made a 
positive difference in learning gains for ABE learners, but not 
for ESL or ASE students; 

• teaching experience ! certification in adult education was 
unrelated to achievement of ESL and ABE clients; it was 
actually negatively related to learning gains of ASE clients 
(that is, those who had less experienced! less qualified 
teachers showed higher gains); 

Support services 

• the provision of support services made a positive difference in 
retention, hut was unrelated to learning gains for all three 
groups; 

Program structure 

• individualized programming appears to he quite helpful for 
ABE clients, hut not for ESL or ASE clients; 

• . the combination of classroom plus lab work as opposed to 

' either i or was associated with higher gains for ABE learners, 
but not for the other groups; 




35 

A Review of Issues and Literatuire Related to Literacy Program Evaluation 
Ontario Literacy Coalition, 1996 



Program costs 



• higher cost was associated with higher gains for ASE and 
ESL learners, hut not for ABE. (The average per hour cost 
was US $4.57; in an Ontario survey conducted a few years 
earlier the average cost per hour was $6.66 in Ontario Basic 
Skills programs offered through colleges, $6.25 in 
Community Literacy groups, and $13. per contact hour in 
Basic Skills in the Workplace programs. See Woods Cordon, 
p.p. 91-92); 

Clients 

• pretest scores, reflecting prior achievement I ability were 
highly predictive of learning gains for all groups; 

• number of years of previous schooling was related to 
achievement for the ESL group only (perhaps reflecting a 
difference between those with and without first language 
literacy); 

• there was a correlation between race and greater gains for 
ESL and ASE clients, hut not ABE (among ESL clients, white 
learners showed somewhat greater gains; among ASE clients, 
Asians showed somewhat higher achievement). 

In fact, according to the large NEAEP data set, only one 
variable (and not a program variable at all) is consistently and 
substantially related to literacy gains for all kinds of clients: 
their prior achievement level. That is, the best predictor of what 
score a learner achieved on the standardized test of basic skills 
given at the end of the instructional period in the NEAEP 
surveyed programs was that learner’s score on the same test 
when s/he entered the course. The pretest score is essentially a 
measure of what students already know and are able to do in 
the basic skills. Those who know the most at entry gain the most 
over the hoxirs of instruction. Those who are at a lower level, 
relative to their peers, will learn less over the same period of 
time. This relationship accounts for almost half of the variance 
in scores on the post tests, but it does not speak to what 
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program cha.racteristi.cs are most effective, or for whom; it 
simply shows the power of a prior condition, over which program 
providers have little or no control.'^ 

This reality has led some writers to suggest that while many 
programs have the goal of serving those most in need, it may 
also be necessary to consider who is most able to benefit. (See 
Fish and Sampson, 1994, p. 6) Programs of relatively short 
diiration should perhaps not aim at those whose disadvantages 
are greatest. 

A recent review of literacy acquisition programs and results in 
developing countries (Abadzi, 1994) cites observation-based 
evidence that classroom processes make a difference: in Nepal, 
the strongest predictors of achievement were instruction-related 
(use of materials at an appropriate level of difficulty), followed 
by teacher-related variables (effective class management, 
effective use of supervisors [supporters and monitors], 
participatory teaching and active adiilt participation, and 
periodic evaluation. Participant-related variables, such as 
interest, motivation, and free time were also important, but less 
so. (See Abadzi, p.p. 9-10) 



^ This finding is similar to what Beder (1991) described in his review of 
studies of the effects of adult literacy programs on basic skills: 

It also seems likely that learning gain is differential, that some types 
of students are more likely to gain than others. The Ohio study, for 
e.xample, shows a significant relationship between high reading and 
mathematics scores and income. Sunilarly, Kent reports that gains in 
TABE scores were influenced by reading level and gender (p. 112). 
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Section Three 



Conclusions^ Recommendations and References 
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3.1 Conclusions and Recommendations 

It is a general truth in education that program evaluation is 
neglected. There are a number of reasons why this is so: the 
difficulty of convincing progrsim providers to take time from 
instruction for evaluation; the difficulties inherent in trying to 
create a comparison group; the attrition of subjects/leamers over 
tim e; and teachersVinstructors’ lack of expertise in evaluation. 

Adult literacy programs add to this set of barriers their own 
particular and difficult obstacles: the open entry/open exit 
character of many programs, making it impossible to set aside a 
specific time for pre-and post-testing; and the reluctance of 
instructors and clients to engage in an activity which smacks of 
past failure for the clients. 

As a resiilt, we know much less than we need to know about 
what makes adult literacy programs effective. Findings to date 
support a very few generalizations: 

Literacy instruction does promote literacy/ 
acquisition, but the number of learners who are 
unsuccessful for whatever reasons, including 
absenteeism, is high; 

It is probably impossible for a learner to 
significantly increase her/his literacy skills without 
a substantial investment of time, amounting to at 
least 50 hours in the best case and over 200 for 
ESL learners in general literacy classes; 

The best way to maximize learning time is to 
extend it, through out-of-class practice, which is 
most likely to occur if the instruction is keyed to 
materials and situations which the client daily 
encounters and needs to master; this is probably 
also a best strategy for preventing forgetting; 
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The most effective instructional methods and 
learning environments are likely different for 
different kinds of learners (ABE, ESL, ASE); 

Offering aUied services, such as counseling, which* 
clients use, will likely help maintain their 
attendance. 

What we do not know makes perhaps a more impressive list: 

What kind of training shotild ABE/ESL/ASE 
instructors/tutors have? 

Under what circumstances does class/group size 
matter to outcomes? 

Are there best instructional methods for different 
groups of learners? 

Given that we know that there ai'e many, many 
more potential clients for adtilt literacy classes 
than are ever served, should significant effort be 
put into increased outreach? Or, would there be 
more of an impact by putting limited resources into 
maximizing the effectiveness of instruction for 
those clients who find their way to literacy 
instruction, rather than expending efforts and 
moneys on outreach to the majority of low literate 
adults , who are unserved? 

How can general literacy programs be made to fit 
clients’ daily lives such that their application 
increases both motivation and practice? 

Or, should more contextualized (workplace, family, 
other) literacy programs be offered, and fewer 
general programs (with the exception of those 
which are targeted at a secondary diploma or 
equivalent)? 

Given that it is apparently possible to accomplish 
more in less time in these contextualized programs, 
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does that merit making some targeted decisions 
about where to put resources? Or, would doing so 
tend to deprive potential learners of access to 
general literacy programs that might come closer to 
meeting their general life goals? Would offering 
more contextualized classes also satisfy Beder’s 
(1991) recommendation that; 

In order to serve a greater proportion of the 
adult literacy target population, recruitment and 
instruction should be tailored to specific client 
group needs, wants, and motivations (p. 145). 

The issue of program evaluation by itself raises important 
questions. It is highly unlikely that the concern for outcomes- 
based evaluation will weaken, given increasingly scarce 
resources. Nor, we argue, should it: how else can we begin to 
find answers to the important questions listed above? But how 
to go about it is a major question by itself Both the available 
literature and simple logic suggest that isolated program 
evaluation— each program doing its own thing (or nothing)— is 
not the way to arrive at answers to questions that all literacy 
providers have. It is essential to be able to compare programs, 
by having some common evaluation tools. Program providers 
often and quite understandably fear such comparisons, which 
can be used to end programs inappropriately; too soon, on the 
basis of too little data. But the best (and probably the most 
likely) use of common evaluative data is not to terminate, but to 
improve programs. 



A recent initiative in Ontario, the Recogmtion of Adult Literacy 
program (RAL), is an attempt to organize a method for assessing 
individual learner progress uniformly in all adult literacy 
programs in Ontario, making movement from one to another 
easy for clients, and giving recognition (in the form of 
certificates) for achievement of each of a number of levels of 
literacy. 

Such a system requires systematic and uniform assessment 
tools. Work is underway on a diagnostic intake instrument, 
which all service providers could use to help place learners. 
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As well, there must be uniform instruments to measure learner 
outcomes; and there is currently discussion of a portfolio of 
learner work, to be judged by trained assessors. The advantages 
of portfoho assessment are many: it siUows for different 
materials, judged important by the learner and by the 
instructor, to be judged on the same criteria; and it becomes a 
portable exhibition which the learner can share with potential 
employers, admissions officers, and others. But the 
disadvantages are also very real: substantial training has 
proven inadequate to insure consistency across assessors. (The 
state of Vermont has had the most experience in this area, and 
is continuing to work to overcome the problems which have so 
far made portfolio assessment unreliable in their educational 
system.) 

Perhaps it is important for policy makers and program 
developers to also give serious consideration to some 
standardized assessment measures, in addition to individual, 
portfolio work. Whether such measures should be existing adult 
literacy tests, such as the TASE or ABLE or, in the case of ESL 
learners, the CASAS; or existing literacy tests developed for 
school age populations, such as the CAT; or whether they should 
be generic tools like cloze tests which can be adapted to different 
materials which are graded by level of difficulty, so that they 
more closely resemble the content of the program, and function 
as competency as well as general literacy measures, are 
questions that can be addressed once the decision is made to 
include some such measures. Other questions might include the 
frequency with which such assessments are done; are time 
intervals established in advance (e.g. every 25 hours), or is 
assessment done when the client and/or instructor feels a level 
has been reached? 

The biggest question that can be asked about adult literacy 
programs at the local, provincial, or national level is: are these 
programs succeeding in lowering the number of adults who have 
insiifficient Hteracy skills? Ultimately, the goal of adult literacy 
and basic education must be to self-destruct; to become 
redundant. While that is perhaps less realistic wherever 
immigration is a major factor contributing to literacy rates, it is 
nonetheless the large rod against which success must be 
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measured. When the periodic Statistics Canada surveys are 
done, do we see a trend toward fewer and fewer adults at Levels 
1 and 2, and an increasingly large majority at Level 4? While 
such questions must seem very remote to individual service 
providers, it is unwise ever to forget what all the effort is finally 
about. 

There are an extraordinary number of goals and achievements 
of adult hteracy programs. They can and do alter self-concept, 
create helping networks, reduce absenteeism, and equip people 
for better jobs. But if they do not also improve reading and 
writing skills, they fail to meet their essential mandate. If 
improved self-concept or networking or attendance or job 
qualification were the only or the principal goal of a program, it 
is not inevitable or even clear that literacy instniction would be 
the most efficient means. An interesting case in point is work 
with prison populations (Ross, Fabiano & Ross, 1988), which 
strongly suggests that very important cognitive development 
goals can be met without literacy instruction (and presumably 
without literacy increases, though they were not measured), in a 
low hterate population. 

But if the primary function of literacy programs is to increase 
adult literacy, then it is obviously necessary to seek soimd 
answers to two central questions: what are the best ways to help 
adults become literate and what is the evidence? 
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